Method and device for three-dimensional light detection and ranging (LiDAR), and three-dimensional measuring device thereof

ABSTRACT

A three-dimensional imaging apparatus includes: a housing having a first opening and a second opening; a first imaging device configured to collect a first set of image data, the first imaging device being mounted within the housing and having an optical axis through the first opening of the housing and a first field of view configured with respect to the optical axis; and a second imaging device configured to collect a second set of image data, the second imaging device being mounted within the housing and having a scanning plane through the second opening of the housing and a second field of view configured with respect to the scanning plane. The optical axis of the first imaging device and the scanning plane of the second imaging device form an angle. The first field of view and the second field of view do not overlap.

CROSS-REFERENCE TO RELATED APPLICATION

This patent application claims the benefits of Chinese PatentApplication No. 202021269287.5, filed on Jul. 1, 2020, and ChinesePatent Application No. 202021158157.4, filed on Jun. 19, 2020, which areincorporated by references herein in their entireties.

FIELD

Generally, the present disclosure relates to the field ofthree-dimensional (3D) imaging and, more specifically, to a method anddevice for 3D imaging and measuring based on light detection andranging.

BACKGROUND

Indoor three-dimensional (3D) imaging devices may include structuredlight cameras and light detection and ranging (LiDAR) scanners. Existingstructured light cameras have a short visual distance and largedeviations. Existing LiDAR scanners have a long visual distance up to,for example, 100 meters or even more. Existing LiDAR scanners mayprovide highly accurate but slow scanning. These 3D imaging devices aremostly used for industrial applications. Thus, 3D imaging devices areusually expensive industrial equipment. There are few consumer-level 3Dimaging devices. In addition, existing 3D imaging devices may be bulkyand difficult to manipulate and carry around.

Therefore, there is a need for a 3D imaging device with a compact andlightweight design that provides high efficiency and accuracy. Such a 3Dimaging device may allow large-scale, easy applications in, for example,the real-estate industry.

SUMMARY

In an exemplary embodiment, the present disclosure provides athree-dimensional (3D) imaging apparatus. The three-dimensional imagingapparatus includes: a housing having a first opening and a secondopening; a first imaging device configured to collect a first set ofimage data, the first imaging device being mounted within the housingand having an optical axis through the first opening of the housing anda first field of view configured with respect to the optical axis; and asecond imaging device configured to collect a second set of image data,the second imaging device being mounted within the housing and having ascanning plane through the second opening of the housing and a secondfield of view configured with respect to the scanning plane. The opticalaxis of the first imaging device and the scanning plane of the secondimaging device form an angle. The first field of view of the firstimaging device and the second field of view of the second imaging devicedo not overlap.

The first imaging device is a camera and the second imaging device is alight detection and ranging (LiDAR) scanner.

The optical axis of the first imaging device extends in a firstdirection and the scanning plane of the second imaging device extendsalong a second direction that is perpendicular to the first direction.

The first and second imaging devices are mounted next to each otheralong the second direction within the housing.

The first direction is a horizontal direction and the second directionis a vertical direction.

The angle between the optical axis of the first imaging device and thescanning plane of the second imaging device is between 70° and 110°.

The angle between the optical axis of the first imaging device and thescanning plane of second imaging device is 90°.

The housing has a first slanted surface formed on a first side of thefirst opening and a second slanted surface formed on a second side ofthe first opening, wherein the first slanted surface and the secondslanted surface form an angle of 150°.

The second opening is configured around the second imaging device anddefines a maximum angle of 150° for the second field of view of thesecond imaging device in scanning plane of the second imaging device.

The second field of view includes the second direction.

The three-dimensional imaging apparatus further includes: a first motorhaving a first rotational axis along the second direction and configuredto rotate the first and second imaging devices.

The three-dimensional imaging apparatus further includes: acomputer-readable medium having computer-executable instructions storedthereon and a processor configured to execute the computer-executableinstructions to carry out: controlling the first motor to rotate thefirst and second imaging devices a first angle about the firstrotational axis; controlling the first and second imaging devices tocollect the first and second sets of image data; and merging the firstand second sets of imaging data according to the angle between theoptical axis of the first imaging device and the scanning plane of thesecond imaging device. The first angle is between 0° and 360°.

The three-dimensional imaging apparatus further includes: a second motorhaving a second rotational axis along a third direction and configuredto rotate the first and second imaging devices about the secondrotational axis, the third direction being perpendicular to the firstand second directions.

The processor is configured to execute the computer-executableinstructions to control the second motor to rotate the first and secondimaging devices a second angle about the second rotational axis, whereinthe second angle is between 0° and 360°.

In another exemplary embodiment, the present disclosure provides amethod of controlling a three-dimensional imaging apparatus having afirst imaging device and a second imaging device disposed therein. Themethod includes: rotating the first and second imaging devices by afirst motor about a first rotational axis, the first imaging devicehaving an optical axis extending along a first direction, the secondimaging device having a scanning plane extending along a seconddirection, the optical axis of the first imaging device and the scanningplane of the second imaging device forming a first angle between 0° and360; starting the first and second imaging devices to collect a firstand second sets of image data, respectively; receiving the first set ofimage data from the first imaging device and the second set of imagedata from the second imaging device; and merging the first and secondsets of image data according in part to the angle between the opticalaxis of the first imaging device and the scanning plane of the secondimaging device.

The method further includes: a) sending an instruction to the firstmotor indicating a first expected rotational angle; b) rotating thefirst and second imaging devices from a first position to a secondposition according to the instruction; c) determining a first rotationalangle between the first position and the second position; and d)determining a first error between the first expected rotational angleand the first rotational angle.

The method further includes: carrying out a)-d) for a plurality ofiterations; and determining a statistical distribution of the firsterror.

The method further includes: determining a first coordinate of acalibration object based on at least one of the first or second sets ofimage data when the first and second imaging devices are at the firstposition; determining a second coordinate of the calibration objectbased on at least one of the first or second sets of image data when thefirst and second imaging devices are at the second position; anddetermining the first rotational angle based on the first coordinate andthe second coordinate of the calibration object.

The three-dimensional imaging apparatus further includes a second motorconfigured to rotate the first and second imaging devices for a secondangle about a third axis that is perpendicular to the first and secondaxes. The method further includes: e) sending an instruction to thesecond motor indicating a second expected rotational angle; f) rotatingthe first and second imaging devices from a third position to a fourthposition according to the instruction; g) determining a secondrotational angle between the third position and the fourth position; andh) determining a second error between the second expected rotationalangle and the second rotational angle. The second angle is between 0°and 360°.

The method further includes: carrying out e)-h) for a plurality ofiterations; and determining a statistical distribution of the seconderror.

The method further includes: determining a third coordinate of acalibration object based on at least one of the first or second sets ofimage data when the first and second imaging devices are at the thirdposition; determining a fourth coordinate of the calibration objectbased on at least one of the first or second sets of image data when thefirst and second imaging devices are at the fourth position; anddetermining the second rotational angle based on the third coordinateand the fourth coordinate of the calibration object.

BRIEF DESCRIPTION OF THE DRAWING(S)

FIG. 1 is a schematic diagram of a three-dimensional imaging apparatusaccording to an exemplary embodiment of the present disclosure;

FIG. 2 is a schematic top view of the three-dimensional imagingapparatus according to an exemplary embodiment of the presentdisclosure;

FIG. 3 is a schematic side view of the three-dimensional imagingapparatus according to an exemplary embodiment of the presentdisclosure;

FIG. 4 is a schematic diagram of the internal components of thethree-dimensional imaging apparatus according to an exemplary embodimentof the present disclosure, in which the housing of the three-dimensionalimaging apparatus is removed;

FIG. 5 is a schematic diagram of the internal components of thethree-dimensional imaging apparatus according to an exemplary embodimentof the present disclosure, in which the housing is partially removed;

FIG. 6 is a schematic diagram of the three-dimensional imaging apparatushaving a function as a three-dimensional measuring device according toan exemplary embodiment of the present disclosure;

FIG. 7 is a schematic diagram of another aspect of the three-dimensionalimaging apparatus having a function as a three-dimensional measuringdevice according to an exemplary embodiment of the present disclosure;

FIG. 8 is a schematic plan view of the three-dimensional imagingapparatus having a function as a three-dimensional measuring deviceshown in FIG. 7 according to an exemplary embodiment of the presentdisclosure;

FIG. 9 is a schematic side view of the three-dimensional imagingapparatus having a function as a three-dimensional measuring deviceshown in FIG. 7 according to an exemplary embodiment of the presentdisclosure;

FIG. 10 is an exemplary flowchart depicting a process for operating thethree-dimensional imaging apparatus according to an exemplary embodimentof the present disclosure; and

FIG. 11 is an exemplary flowchart depicting a process for operating thethree-dimensional imaging apparatus according to an exemplary embodimentof the present disclosure.

DETAILED DESCRIPTION

Exemplary embodiments of the present disclosure provide athree-dimensional (3D) imaging apparatus based on light detection andranging (LiDAR) that may be used for consumer applications on a largescale. The three-dimensional imaging apparatus may include a LiDARscanner and a camera with a compact and lightweight design. Thethree-dimensional imaging apparatus may also provide high efficientimaging and measuring. The three-dimensional imaging apparatus disclosedherein may be much less costly and more user-friendly than existingthree-dimensional imaging devices. The three-dimensional imagingapparatus disclosed herein may enjoy a wide variety of applicationsincluding, but not limited to, real-estate applications, such as imagingand measuring real properties.

According to some embodiments, the 3D imaging apparatus may include afirst imaging device and a second imaging device integrated in ahousing. The first imaging device may be a camera, and the secondimaging device may be a LiDAR scanner. The camera may have a lenssystem. The lens system may define an optical axis and a field of view.The camera may receive light from a 3D scene through the lens system andgenerate images of the 3D scene based on the received light.

The LiDAR scanner may have a transmitter that generates and transmitsstructured light, such as laser, to the 3D scene and a receiver thatreceives portion of the structured light reflected by the 3D scene. TheLiDAR device may provide a scanning function by moving the structuredlight through the 3D scene according to a defined direction, such as avertical direction or a horizontal direction. The reflected structuredlight received during the scanning of the 3D scene may be used to form a3D model of the 3D scene. The moving structured light may define ascanning plane of the LiDAR scanner that intercepts with the 3D scene.The 3D model of the 3D scene may be reconstructed or generated accordingto the scanning plane. The structured light moving within the scanningplane may form a field of view defined by an angle.

According to some embodiments, the first imaging device and the secondimaging device are arranged in such a manner so that the field of viewof the first imaging device and the field of view of the second imagingdevice do not overlap each other. For example, the first imaging deviceand the second imaging device may be staggered so that the optical axisof the first imaging device and the scanning plane of the second imagingdevice form an angle.

According to some embodiments, the 3D imaging apparatus may furtherinclude a first motor that rotates the first and second imaging devicesabout a first axis so that the first and second imaging devices mayperform a scanning of the 3D scene about the first axis. The 3D imagingapparatus may further include a second motor that rotates the first andsecond imaging devices about a second axis so that the first and secondimaging devices may perform a scanning of the 3D scene about the secondaxis.

According to some embodiments, the 3D imaging apparatus may becalibrated using images of the 3D scene. For example, when the 3Dimaging apparatus is at a first position, a first image may be taken. Afirst signal may be sent to the first motor indicating a first targetrotational angle. Upon receiving the first signal, the first motor mayrotate the 3D imaging apparatus by a first rotational angle from thefirst positional to a second position. A second image may be taken bythe 3D imaging apparatus when it is at the second position. The firstimage and second image may be compared to determine the first rotationalangle. A difference between the first rotational angle and the firsttarget rotational angle indicated by the first signal may be determinedand used to calibrate the first motor. The second motor may becalibrated in a likewise manner. Thus, the quality of the data collectedby the 3D imaging apparatus may be improved.

FIGS. 1-3 show schematic perspective views of a three-dimensional (3D)imaging apparatus 105 according to some embodiments of the presentdisclosure. As shown in the figures, the 3D imaging apparatus 105 may beoriented in the 3D scene as desired. For ease of discussion, a verticaldirection VV′ may be indicated by an arrow VV′ as shown in FIGS. 1 and 3according to an embodiment.

3D imaging apparatus 105 may include a housing 100, a first imagingdevice 200, and a second imaging device 300. Housing 100 may include afirst opening 110 (shown in FIG. 1 ) and a second opening 140 (shown inFIGS. 2 and 3 ), according to some embodiments.

First imaging device 200 may be configured to generate a first set ofimage data. First imaging device 200 may operate according to principlesknown in the art. For example, First imaging device 200 may be anoptical camera, a heat detector, an X-ray detector, a Gamma-raydetector, etc. Thus, the first set of image data may be two-dimensionalimages including information imparted onto a sensor of first imagingdevice 200 according to one or more of those known principles.

FIGS. 4 and 5 show schematic perspective view of the internal componentsof the 3D imaging apparatus 105 according to some embodiments of thepresent disclosure, where the housing 100 is removed to reveal internalstructures. According to an embodiment, first image device 200 mayinclude a lens system which has an optical axis as indicated by a XX′line shown in FIGS. 4 and 5 . The lens system of first imaging device200 has a field of view that may be defined or configured with respectto the optical axis XX′. The field of view of first imaging device 200may be, for example, a cone-shaped field of view (CVF) as shown in FIG.4 . The optical axis XX′ may be the axis of the cone-shaped field ofview. The field of view of first imaging device 200 may also have othergeometrical shape, such as a cylindrical shape, a fan shape, etc.

According to some embodiments, first imaging device 200 may collectimages of the surrounding environment (e.g., the 3D scene) that fallswithin the field of view of first imaging device 200. The images of thesurrounding environment collected by first imaging device 200 may bereferred to as environmental images, which may reflect the environment,appearance of the environment, and/or appearance of an object of theenvironment. According to an embodiment, the images collected by firstimaging device 200 may be two-dimensional images that may be colorimages, black-and-white images, or grayscale images.

In some embodiments, as shown in FIG. 1 , housing 100 of 3D imaginingapparatus 105 may have a first opening 110. First imaging device 200 maybe disposed within housing 100 so that the lens system of first imagingdevice 200 is substantially close to and aligned with first opening 110.Thus, the optical axis XX′ of first imaging device 200 may go throughthe first opening 110 and be aligned with the axis of first opening 110.The field of view of first imaging device 200 may extend through firstopening 110.

According to some embodiments, second imaging device 300 may beconfigured to scan and generate three-dimensional information of theenvironment or the scene. For example, second imaging device 300 may bea scanner that operates according to the light detection and ranging(LiDAR) principle known in the art. Specifically, second imaging device300 may include a laser transceiver that emits laser signals towards the3D scene and receive returned laser signals from the 3D scene. Secondimaging device 300 may include a processor configured to generate 3Dinformation of the scene based on the returned laser signals.Alternatively, second imaging device 300 may transmit data representingthe returned laser signals to an off-board processor, which may generatethe 3D information of the scene. The 3D information of the scene may bea 3D model including 3D coordinate data that represent features of the3D scene.

According to a further embodiment as shown in FIGS. 4 and 5 , secondimaging device 300 may have one scanning plane (SP). The laser signalsemitted by second imaging device 300 may be kept within the scanningplane. Specifically, the laser signals emitted by second imaging device300 may include a laser beam that is moved within the scanning plane.Scanning of the 3D scene may be conducted by rotating or swiping thelaser beam within the scanning plane across a field of view (RVF) ofsecond imaging device 300. The field of view of second imaging device300 may be defined by an angle a as shown in FIG. 5 . According to someembodiments, second imaging device 300 may have more than one scanningplane.

According to some embodiments, housing 100 of the 3D imaging apparatus105 may have a second opening 140 as shown in FIGS. 2, 3, and 5 . Thescanning plane and the field of view of second imaging device 300 mayextend through second opening 140.

According to some embodiments, second imaging device 300 may determine3D coordinates of a feature point of the 3D scene by comparingparameters of the emitted laser signals and the returned laser signals.The coordinates of the feature point determined by second imaging device300 fall on the intersection between the 3D scene and the scanning planeof second imaging device 300. As such, second imaging device 300collects the coordinates of a plurality of points of the 3D scene thatfall on the scanning plane. These points contain 3D information of the3D scene and may form a point cloud that represents a 3D structuralmodel of the 3D scene.

According to an embodiment, the 2D images collected by first imagingdevice 200 represent an appearance of the 3D scene, such as the colorsor textures of the objects of the 3D scene. According to an embodiment,data collected by second imaging device 300 may include 3D informationof the 3D scene. The 3D information may be a 3D point cloud thatrepresents, for example, a structural model of the 3D scene. Thestructural model of the 3D scene may be combined with the 2D imagescollected by first imaging device 200 to generate a 3D virtual realityrepresentation of the 3D scene. For example, the 2D images collected byfirst imaging device 200 may be mapped to the 3D model generated bysecond imaging device 300 based on position, orientation, and/distanceinformation included in the data collected by first imaging device 200and second imagine device 300. The resulting 3D virtual realityrepresentation of the 3D scene may include the 3D model modified by thecolor information of the 2D images.

According to a further embodiment, as shown in FIGS. 4 and 5 , theoptical axis (indicated by line XX′) of first imaging device 200 extendsin a first direction and the scanning plane of second imaging device 300extends along a second direction that is perpendicular to the firstdirection. First imaging device 200 and second imaging device 300 aremounted next to each other along the second direction within housing 100of 3D imaging apparatus 105. According to a further embodiment, thefirst direction is a horizontal direction and the second direction is avertical direction.

According to a further embodiment as shown in FIG. 5 , the optical axisXX′ of first imaging device 200 and the scanning plane of second imagingdevice 300 may form an angle ∅. The angle ∅ may be sufficiently large sothat the field of view of first imaging device 200 and the field of viewof second imaging device 300 are outside of each other. In other words,the field of view of first imaging device 200 and the field of view ofsecond imaging device 300 do not overlap each other. Accordingly, firstimaging device 200 is not located on the scanning plane of secondimaging device 300. And second imaging device 300 is not located in thefield of view of first imaging device 200. As a result, data collectedby second imaging device 300 does not include an image of first imagingdevice 200. Similarly, data collected by first imaging device 200 doesnot include an image of second imaging device 300.

As shown in FIGS. 4 and 5 , because first imaging device 200 and secondimaging device 300 are arranged in such a way that their fields of viewdo not overlap each other, first imaging device 200 and second imagingdevice 300 may be disposed as close as possible to each other and fit ina compact structure within housing 100 of 3D imaging apparatus 105. Thisarrangement of first imaging device 200 and second imaging device 300may decrease the size of 3D imaging apparatus 105 compared with those ofconventional designs. Thus, 3D imaging apparatus 105 may be used toperform three-dimensional modeling of a 3D scene in a small space, suchas an interior of a house, a room, an office, or other settings. 3Dimaging apparatus 105 described herein may replace existingthree-dimensional imaging devices, which are large and bulky and mayonly be used for modeling of a large-scale space. In addition, becauseof the compact structure of 3D imaging apparatus 105, it may be easilystored and carried around by a user.

According to a further embodiment, second imaging device 300 may bemounted above first imaging device 200 within housing 100 in thevertical direction as shown in FIGS. 4 and 5 . Alternatively, secondimaging device 300 may be mounted under first imaging device 200 withinhousing 100 in the vertical direction VV′. Still further, the bodystructures of second imaging device 300 and first imaging device 200 maybe disposed as close to each other as possible and may partially overlapeach other along the vertical direction VV′.

Such arrangement of first imaging device 200 and second imaging device300 reduces not only the size or footprint of 3D imaging apparatus 105in the horizontal direction, but also the size or footprint of 3Dimaging apparatus 105 in the vertical direction.

According to some embodiments, the angle ∅ between the optical axis XX′of first imaging device 200 and the scanning plane of second imagingdevice 300 may be between 70° and 110°. According to some furtherembodiments, the angle ∅ may be between 80° and 100°. According to stillsome further embodiment, the angle ∅ may be about 90°.

These arrangements of first imaging device 200 and second imaging device300 may avoid interferences between the two imaging devices and reducethe overall size and footprint of 3D imaging apparatus 105. In addition,these arrangements of first imaging device 200 and second imaging device300 may also provide a large viewing angle along the vertical directionVV′ for both imaging devices. As such, 3D imaging apparatus 105 may beable to collect more comprehensive data and generate more completethree-dimensional models of the 3D scene.

According to a further embodiment, housing 100 of 3D imaging apparatus105, as shown in FIG. 1 , may include a pair of slanted outer surfaces120 and 130, which are formed on a first side and a second side of firstopening 110. Specifically, as shown in FIG. 1 , first slanted outersurface 120 may be formed on an upper side (i.e., the first side) offirst opening 110. Second slanted outer surface 130 may be formed on asecond side (i.e., the second side) of first opening 110. Alternatively,slanted surfaces 120 and 130 may be formed at other locations near firstopening 110. For example, slanted surfaces 120 and 130 may be formed onthe left and right sides of first opening 110, respectively. Accordingto a further embodiment, slanted surfaces 120 and 130 may be flat orcurved. According to a still further embodiment, slanted surfaces 120and 130 may be formed and arranged symmetrically with respect to theoptical axis XX′ of first imaging device 200, as shown in FIG. 1 .

The pair of slanted outer surfaces 120 and 130 may form an angle ∅between each other. In a further embodiment, the angle ∅ between slantedsurfaces 120 and 130 may be at least 150°. As such, slanted surfaces 120and 130 define a limit of the field of view of first imaging device 200along the vertical direction VV′ (or any direction along which slantedsurfaces 120 and 130 are disposed).

According to a further embodiment as shown in FIGS. 2, 3, and 5 ,housing 100 of 3D imaging apparatus 105 may include a second opening140. The scanning plane of second imaging device 300, on which the fieldof view of second imaging device 300 is defined, may pass through secondopening 140. The scanning plane of second imaging device 300 may extendalong a second direction, such as the vertical direction VV′, and passthrough second opening 140.

According to a further embodiment, second opening 140 may be configuredaround second imaging device 300 and defines a maximum angle for thefield of view of second imaging device 300. The maximum angle defined bysecond opening 140 may be at least 150°.

Through second opening 140, second imaging device 300 may transmit andreceive signals, such as laser signals, within the maximum angle of thefield of view in the scanning plane. In other words, second imagingdevice 300 may be configured to scan the 3D scene through second opening140. The scanning may be done within the maximum angle of the field ofview described above. Based on the data collected by scanning the 3Dscene within the maximum angle of field of view, second imaging device300 may generate a 3D model, which includes a point cloud representing3D structural information of the 3D scene. As one of ordinary skilled inthe art may appreciate, the larger the maximum angle of the field ofview of second imaging device 300 provided by second opening 140, themore complete the 3D model generated by second imaging device 300 forthe 3D scene.

According to a further embodiment as shown in in FIGS. 2, 3, and 5 ,second opening 140 may extends around second imaging device 300 beyond apoint directly above second imaging device 300. As such, second imagingdevice 300 may be configured to transmit and receive scanning signalsdirectly above second imaging device 300, thereby generating acomprehensive 3D model of the 3D scene.

According to a further embodiment, as shown in FIGS. 4 and 5 , 3Dimaging apparatus 105 may further include a motor 400 configured todrive first imaging device 200, second imaging device 300, and housing100 (when installed) to rotate about a first axis, i.e., the rotationalaxis of motor 400. The rotational axis of motor 400, as shown in FIGS. 4and 5 , may be along the vertical direction VV′, so that first imagingdevice 200 and second imaging device 300 may be rotated in a horizontalplane. Specifically, motor 400 may receive control signals from acontrol module (not shown) disposed within housing 100. In response tothe control signals, motor 400 may rotate continuously or make one ormore stops during rotation. First imaging device 200 and second imagingdevice 300 may receive control signals from the control module. Inresponse to the control signals, imaging device 200 and second imagingdevice 300 may be activated to collect data of the 3D scene. Forexample, motor 400 may make a stop after rotating to a first angle inresponse to the control signals. While motor 400 stops, first imagingdevice 200 may be activated to capture an image, such as a 2D image, ofthe 3D scene, and second imaging device 300 may be activated to conducta scanning of the 3D scene. After first imaging device 200 and secondimaging device 300 complete the data collection, motor 400 may rotate toa second angle in response to additional control signals and stop at thesecond angle. First imaging device 200 and second imaging device 300 maybe activated to capture additional data of the 3D scene. The process maythen be repeated until motor 400 rotates for 360° or a desired amount ofdata of the 3D scene is collected.

Thereafter, all of the data collected by first imaging device 200 andsecond imaging device 300 may be combined to generate a virtual reality3D model of the 3D scene. Specifically, the 2D images collected by firstimaging device 200 may be combined to form a panoramic image of the 3Dscene. The data collected by second imaging device 300 may be used togenerate a point cloud representing the 3D model of the 3D scene. Thepanoramic image of the 3D scene may then be mapped to the 3D model ofthe 3D scene based on the relative positions and orientations of firstimaging device 200 and second imaging device 300.

According to a further embodiment, as shown in FIGS. 4 and 5 , 3Dimaging apparatus 105 further includes a frame 500 that is mountedwithin housing 100. First imaging device 200, second imaging device 300,and motor 400 may be secured to frame 500. Frame 500 allows thepositions and orientations of first imaging device 200, second imagingdevice 300, and motor 400 to be adjusted easily. Frame 500 furtherallows first imaging device 200, second imaging device 300, and motor400 to be easily calibrated at the production stage of 3D imagingapparatus 105. After the calibration is completed, first imaging device200, second imaging device 300, and motor 400 may be affixed to frame500 to form an assembly, which may be easily disposed within housing 100while the relative positions and orientations of the components thereinare maintained during the production stage.

The calibration of first imaging device may include determining internalparameters of first imaging device 200, such as focal length, imagecenter, distortions, etc., determining relative positions andorientations of first imaging device 200 and second imaging device 300,the orientation of the rotational axial of motor 400. Propercalibrations may be needed for generating high-quality virtual reality3D model of the 3D scene.

In some exemplary embodiments, 3D imaging apparatus 105 may be set up ona tripod. Motor 400 is configured to drive 3D imaging apparatus 105,first imaging device 200 and second imaging device 300, to rotatehorizontally in relation to the tripod. As such, 3D imaging apparatus105 may be used to collect data of the 3D scene and generate a panoramic3D model of the 3D scene. One of ordinary skilled in the art mayappreciate that motor 400 may also be arranged outside of housing 100directly on the tripod instead of being mounted within housing 100 of 3Dimaging apparatus 105.

In some exemplary embodiments, 3D imaging apparatus 105 may include asecond motor configured to rotate 3D imaging apparatus 105 or one offirst imaging device 200 and second imaging device 300 individuallyabout a second axis. The second axis may be perpendicular to the firstaxis (i.e., the rotational axis of motor 400). Specifically, FIGS. 6-9show an embodiment of a 3D imaging apparatus 10 that is similar to 3Dimaging apparatus 105 disclosed above.

3D imaging apparatus 10 may include a base 1 and a first motor 2disposed on base 1. First motor 2 is similar to motor 400 as shown inFIGS. 4 and 5 and configured to rotate a frame 3 of 3D imaging apparatus10. One of ordinary skilled in the art may appreciate that frame 3 maybe similar to frame 500 as shown in FIGS. 4 and 5 . 3D imaging apparatus10 may include an imaging device 5 disposed on frame 3. One of ordinaryskilled in the art may appreciate that imaging device 5 may be firstimagining device 200, second imaging device 300, or a combination offirst imaging device 200 and second imaging device 300 as shown in FIGS.4 and 5 .

First motor 2 may be configured to rotate in accordance with a firstrotation signal from a control module (not shown) disposed within 3Dimaging apparatus 10. The first rotation signal may indicate a firstexpected rotational angle set for motor 2. In some exemplaryembodiments, first motor 2 may be a servo motor or a step motor. Inresponse to the first rotation signal, motor 2 may drive frame 3 andimaging device 5 to rotate about a first axis (e.g., axis ZZ′) from afirst initial position to a first target position according to the firstexpected rotational angle. When motor 2 stops at the first targetposition, imaging device 5 may be triggered to capture a set of data ofthe 3D scene.

According to a further embodiment, first motor 2 may rotate and stop formultiple times based on the first rotation signal. At each stop, imagingdevice 5 may be activated to collect the data of the 3D scene. Forexample, imaging device 5 may be activated to capture a 2D image and/orto conduct a scanning of the 3D scene. First motor 2 may rotate for 360°or until a desired amount of data is collected.

According to some embodiments, 3D imaging apparatus 10 may furtherinclude a second motor 4 that is arranged on a vertical supporting sideof frame 3, as shown in FIGS. 7 and 9 . Second motor 4 may be configuredto rotate imaging device 5 about a second axis (e.g., axis YY′) inaccordance with a second rotation signal from the control module (notshown) disposed within the 3D imaging apparatus 10. In an embodiment,the second axis may be perpendicular to the first axis. Accordingly,second motor 4 may generate a pitch motion for imaging device 5 inaddition to the rotational motion generated by first motor 2. In someexemplary embodiments, second motor 4 may be a servo motor or a stepmotor.

According to a further embodiment, the second rotation signal mayindicate a second expected rotational angle. In response to the secondrotation signal, second motor 4 may rotate imaging device 5 around thesecond axis from a second initial position to a second target positionaccording to the second expected rotational angle. When second motor 4stops at the second target position, imaging device 5 may be triggeredto capture a set of data of the 3D scene.

According to a further embodiment, second motor 4 may rotate and stopfor multiple times based on the second rotation signal. At each stop,imaging device 5 may be activated to collect a set of data of the 3Dscene. For example, imaging device 5 may be activated to capture a 2Dimage and/or to conduct a scanning of the 3D scene. Second motor 4 mayrotate for 360° or until a desired amount of data is collected.

According to a further embodiment, as shown in FIGS. 6-9 , frame 3 mayhave a U shape, the bottom of which is fixed to first motor 2. The twovertical supporting sides of frame 3 may be fixed to the opposite sidesof imaging device 5 through second motor 4.

In some exemplary embodiments, 3D imaging apparatus 10 may furtherinclude a detection circuit 6 electrically connected to first motor 2,second motor 4, and imaging device 5. Detection circuit 6 may beconfigured to detect and compare the rotations of motors 2 and 4 withthe expected rotational angles indicated by the first rotation signalsand the second rotation signals. Based on the comparisons, detectioncircuit 6 may detect errors in the motions of motors 2 and 4.

For example, the angle between the first initial position and the firsttarget position of first motor 2 may be different from the firstexpected rotational angle indicated by the first rotation signalreceived by first motor 2. Similarly, the angle between the secondinitial position and the second target position of second motor 4 may bedifferent from the second expected rotational angel indicated by thesecond rotation signal received by second motor 4. In other words, theremay be errors in the rotations of first motor 2 and second motor 4 dueto, for example, mechanical or electrical variations. Thus, it isdesired to determine the errors in the rotations of first motor 2 andsecond motor 4 so as to eliminate or reduce the errors when the datacollected by 3D imaging apparatus 10 is used to build the 3D virtualreality model of the 3D scene.

FIG. 10 illustrates a process 1000 for generating a 3D virtual realitymodel of a 3D scene using a 3D imaging apparatus, such as 3D imagingapparatus 10 and 3D imaging apparatus 105 disclosed above, according toan embodiment. Process 1000 may be carried out by a processor accordingto instructions stored on a computer-readable medium. The processor andthe computer-readable medium may be disposed on a control module, suchas detection circuit 6 or similar circuitry that is electricallyconnected with the 3D imaging apparatus.

According to process 1000, at step 1002, first imaging device 200 andsecond imaging device 300 are rotated by motor 400 or motor 2 (i.e., thefirst motor) about a first rotational axis. According to an embodiment,the first rotational axis may be the rotational axis of the first motor.The first rotational axis may be along a vertical direction VV′ as shownin FIGS. 3 and 4 .

At step 1004, first imaging device 200 and second imaging device 300 maybe started to collect a first set of image data and the second set ofimage data of the 3D scene, respectively. According to an embodiment,first imaging device 200 may be a camera that collects and generatestwo-dimensional images (i.e., the first set of image data) of the 3Dscene. According to another embodiment, second imaging device 300 may bea laser ranger or a LiDAR device, as known in the art, which scans the3D scene using light signals. Based on the light signals, second imagingdevice 300 may generate 3D structural data (i.e., the second set ofimage data) of the 3D scene, such as a point cloud. First imaging device200 and second imaging deice 300 may also use other type of signals,such as radar signals, infrared signals, to generate the first andsecond sets of image data.

At step 1006, the first and second sets of image data may be receivedfrom first imaging device 200 and second imaging device 300. Accordingto an embodiment, the first and second sets of image data may bereceived by an on-board processing unit or an off-board processing unit.The first and second sets of image data may be stored by acomputer-readable storage medium for later processing.

At step 1008, the first and second sets of image data may be merged toform a 3D virtual reality model of the 3D scene. According to anembodiment, the first set of image data may include one or more 2Dimages generated by first imaging device 200. The 2D images may form apanoramic image of the 3D scene. For example, first imaging device 200may be started to collect a series of 2D images while being rotated bythe first motor about the first rotational axis. The series of 2D imagesmay be merged to form the panoramic image of the 3D scene.

According to an embodiment, the second set of image data may include a3D structural model of the 3D scene. The 3D structural model may includea 3D point cloud generated based on the scanning of the 3D sceneperformed by second imaging device 300. The 3D point cloud may includestructural information indicating the 3D coordinates of features of the3D scene.

According to an embodiment, the first set of image data and the secondset of image data may be merged based on the 3D relationship betweenfirst imaging device 200 and second imaging device 300. For example, afirst coordinate system may be defined for first imaging device 200based on the location and position of first imaging device 200. Thefirst set of image data may be associated with the first coordinatesystem based on the intrinsic parameters of first imaging device 200.Similarly, a second coordinate system may be defined for second imagingdevice 300 based on the location and position of second imaging device300. The second set of image data may be associated with the secondcoordinate system based on the intrinsic parameters of second imagingdevice 300.

According to an embodiment, the first and second sets of image data maybe merged by converting the first set of image data from the firstcoordinate system to the second coordinate system based on the anglebetween the optical axis of first imaging device 200 and the scanningplane of second imaging device 300. Alternatively, the first and secondsets of image data may be merged by converting the second set of imagedata from the second coordinate system to the first coordinate systembased on the angle between the optical axis of first imaging device 200and the scanning plane of second imaging device 300. After theconversion, the first set of image data (i.e., the panoramic image) maybe mapped to the second set of image data (i.e., the 3D structuralmodel) to generate the 3D virtual reality model of the 3D scene. The 3Dvirtual reality model may include a 3D representation of the 3D scenesuch as structures, textures, colors of features within the 3D scene.

According to a further embodiment, steps 1002 and 1004 may be performedalternatively or at the same time. For example, the first motor may makeone or more stops during the rotation of the first and second imagingdevices at step 1002. When the first motor make a stop, step 1004 may beperformed, at which the first and second imaging devices may be startedto take the first and second sets of image data. After the first andsecond imaging devices complete the acquisition of the image data,process 1000 may return to step 1002, at which the first motor may beactivated to continue rotating the first and second imaging devices. Theprocess may continue until the first motor rotates for 360° or a setangle.

According to a further embodiment, the 3D imaging apparatus may includea second motor (i.e., motor 4 as shown in FIGS. 7 and 9 ), which isconfigured to rotate the first imaging device (or the second imagingdevice) about a second rotational axis (i.e., the rotational axis ofmotor 4). The second rotational axis may be along a third direction(i.e., the YY′ direction as shown in FIG. 7 ) and perpendicular to thefirst rotational axis.

According to an embodiment, process 1000 may further include rotatingthe first imaging device (or the second imaging device) by the secondmotor about the second rotational axis and controlling the second motorto make one or more stops during the rotation of the first imagingdevice (or the second imaging device) about the second rotational axis.For example, when the first motor makes a stop as disclosed above, thesecond motor may rotate and also make a stop. At this time, the firstand second imaging devices may be started to take the first and secondsets of image data. Thereafter, the second motor may be activated andthen make another stop. The first and second imaging devices may bestarted again to take the first and second sets of image data. Theprocess may be continued until the acquisition of the image data iscompleted for the stop of the first motor. Thereafter, the first motoris activated and then make another stop. The steps above may be repeateduntil the first motor rotates for 360° or a set angle.

It will be appreciated that stopping of the first and second motors mayswitched. For example, the second motor may make a stop first. The firstmotor may then rotate and make one or more stops for the first andsecond imaging devices to take the first and second sets of image data.Then the second motor may be activated and make another stop. The stepsmay be repeated until the second motor rotates for 360° or a set angle.

As discussed above, due to the errors in the rotations of the first andsecond motors, the expected rotational angles set for the first andsecond motors may be different from the rotational angles by which thefirst and second motors actually rotate. A calibration is performed onthe 3D imaging apparatus in order to determine the errors in therotations of the first and second motors, according to variousembodiments.

FIG. 11 illustrates a process 1100 for controlling the 3D imagingapparatus to perform a calibration, according to an embodiment. It willbe appreciated that process 1100 may be applied to the calibration ofthe first motor (i.e., motor 400 or motor 2) and the second motor (i.e.,motor 4). The calibration of the first motor will be describedhereinafter for purposes of illustrations. Specifically, whencalibrating the first motor according to process 1100, at step 1102, afirst rotational signal may be sent to the first motor. The firstrotational signal may include an instruction that indicates a firstexpected rotational angle for the first motor. The first rotationalsignal may be sent by a control module on-board or off-board the 3Dimaging apparatus. The first expected rotational angle may be arotational angle set for the first motor by a user or a computer programassociated with the 3D imaging apparatus.

At step 1104, the first and second imaging devices of the 3D imagingapparatus may be rotated by the first motor from a first position to asecond position according to the instruction. At step 1106, a firstrotational angle between the first position and the second position maybe determined. At step 1108, a first error between the first expectedrotational angle and the first rotational angle is determined.

According to an embodiment, the first error determined at step 1108 maybe used as a parameter for processing the first and second sets of imagedata according to process 1000 as shown in FIG. 10 . For example, thefirst error may be compensated or reduced when the panoramic image isformed based on the 2D images taken by the first imaging device and whenthe point cloud is formed based on the 3D data collected by the secondimaging device.

According to a further embodiment, steps 1102-1106 may be performed fora plurality of iterations. Accordingly, a plurality of sample values ofthe first error may be determined. Based on the plurality of samplevalues, processor 1100 may further include determining a statisticaldistribution of the first error. The statistical distribution of thefirst error may be, for example, a normal distribution that is definedby parameters, such as a mean and a standard deviation. The statisticaldistribution may be used as a parameter for processing the first andsecond sets of image data according to process 1000 as shown in FIG. 10. For example, the mean of the first error may be used to compensate orreduce the misalignment when the panoramic image is formed based on the2D images taken by the first imaging device and when the point cloud isformed based on the 3D data collected by the second imaging device.

According to a further embodiment, the first rotational angle of thefirst and second imaging devices disclosed above may be determinedaccording to a calibration object. As shown in FIGS. 6-9 , for example,a calibration object 7 may be disposed within the field of view of thefirst imaging device. When the first and second imaging devices are atthe first position, the first set of image data taken by the firstimaging device may include a first image of calibration object 7. Thefirst image of calibration object 7 may be used to determine a firstcoordinate of calibration object 7.

When the first and second imaging devices are at the second position,the first set of image data taken by the first imaging device mayinclude a second image of calibration object 7. The second image ofcalibration object 7 may be used to determine a second coordinate ofcalibration object 7. The first rotational angle may be determined basedon the first coordinate and the second coordinate of calibration object7 as one of ordinary skilled in the art will appreciate. The first andsecond coordinates disclosed above may be, for example, 3D coordinatessuch as Cartesian coordinates, polar coordinates, etc. Further, thefirst and second coordinates of calibration object 7 may be calculatedaccording to known a technique, such as machine vision using thePerspective-n-Point (PNP) algorithm or other algorithm generally knownin the art.

According to an alternative embodiment, calibration object 7 may bedisposed within field of view of the second imaging device. A processsimilar to that disclosed above may be applied to determine the firstrotational angle. According to still an alternative embodiment, thefirst position, the second position, and the first rotational angle ofthe first and second imaging devices disclosed above may be determinedby other means, such as a positional sensor, a motion sensor, etc., asknown in the art.

One of ordinary skill in the art will appreciate that the calibration ofthe second motor may also be performed according to the similar processas disclosed above. Thus, the calibration of the second motor is notrepeated herein.

All references, including publications, patent applications, andpatents, cited herein are hereby incorporated by reference to the sameextent as if each reference were individually and specifically indicatedto be incorporated by reference and were set forth in its entiretyherein.

The use of the terms “a” and “an” and “the” and “at least one” andsimilar referents in the context of describing the invention (especiallyin the context of the following claims) are to be construed to coverboth the singular and the plural, unless otherwise indicated herein orclearly contradicted by context. The use of the term “at least one”followed by a list of one or more items (for example, “at least one of Aand B”) is to be construed to mean one item selected from the listeditems (A or B) or any combination of two or more of the listed items (Aand B), unless otherwise indicated herein or clearly contradicted bycontext. The terms “comprising,” “having,” “including,” and “containing”are to be construed as open-ended terms (i.e., meaning “including, butnot limited to,”) unless otherwise noted. Recitation of ranges of valuesherein are merely intended to serve as a shorthand method of referringindividually to each separate value falling within the range, unlessotherwise indicated herein, and each separate value is incorporated intothe specification as if it were individually recited herein. All methodsdescribed herein can be performed in any suitable order unless otherwiseindicated herein or otherwise clearly contradicted by context. The useof any and all examples, or exemplary language (e.g., “such as”)provided herein, is intended merely to better illuminate the disclosureand does not pose a limitation on the scope of the disclosure unlessotherwise claimed. No language in the specification should be construedas indicating any non-claimed element as essential to the practice ofthe disclosure.

Exemplary embodiments of this disclosure are described herein, includingthe best mode known to the inventors for carrying out the disclosure.Variations of those exemplary embodiments may become apparent to thoseof ordinary skill in the art upon reading the foregoing description. Theinventors expect skilled artisans to employ such variations asappropriate, and the inventors intend for the disclosure to be practicedotherwise than as specifically described herein. Accordingly, thisdisclosure includes all modifications and equivalents of the subjectmatter recited in the claims appended hereto as permitted by applicablelaw. Moreover, any combination of the above-described elements in allpossible variations thereof is encompassed by the disclosure unlessotherwise indicated herein or otherwise clearly contradicted by context.

What is claimed is:
 1. A three-dimensional imaging apparatus,comprising: a housing having a first opening and a second opening; afirst imaging device configured to collect a first set of image data,the first imaging device being mounted within the housing and having anoptical axis through the first opening of the housing and a first fieldof view configured with respect to the optical axis; and a secondimaging device configured to collect a second set of image data, thesecond imaging device being mounted within the housing and having ascanning plane through the second opening of the housing and a secondfield of view configured with respect to the scanning plane, wherein theoptical axis of the first imaging device and the scanning plane of thesecond imaging device form an angle, and wherein the first field of viewof the first imaging device and the second field of view of the secondimaging device do not overlap.
 2. The three-dimensional imagingapparatus of claim 1, wherein the first imaging device is a camera andthe second imaging device is a light detection and ranging (LiDAR)scanner.
 3. The three-dimensional imaging apparatus of claim 1, whereinthe optical axis of the first imaging device extends in a firstdirection and the scanning plane of the second imaging device extendsalong a second direction that is perpendicular to the first direction.4. The three-dimensional imaging apparatus of claim 3, wherein the firstand second imaging devices are mounted next to each other along thesecond direction within the housing.
 5. The three-dimensional imagingapparatus of claim 3, wherein the first direction is a horizontaldirection and the second direction is a vertical direction.
 6. Thethree-dimensional imaging apparatus of claim 1, wherein the anglebetween the optical axis of the first imaging device and the scanningplane of the second imaging device is between 70° and 110°.
 7. Thethree-dimensional imaging apparatus of claim 6, wherein the anglebetween the optical axis of the first imaging device and the scanningplane of second imaging device is 90°.
 8. The three-dimensional imagingapparatus of claim 1, wherein the housing has a first slanted surfaceformed on a first side of the first opening and a second slanted surfaceformed on a second side of the first opening, wherein the first slantedsurface and the second slanted surface form an angle of 150°.
 9. Thethree-dimensional imaging apparatus of claim 3, wherein the secondopening is configured around the second imaging device and defines amaximum angle of 150° for the second field of view of the second imagingdevice in scanning plane of the second imaging device.
 10. Thethree-dimensional imaging apparatus of claim 9, wherein the second fieldof view includes the second direction.
 11. The three-dimensional imagingapparatus of claim 3, further comprising a first motor having a firstrotational axis along the second direction and configured to rotate thefirst and second imaging devices.
 12. The three-dimensional imagingapparatus of claim 11, further comprising a computer-readable mediumhaving computer-executable instructions stored thereon and a processorconfigured to execute the computer-executable instructions to carry out:controlling the first motor to rotate the first and second imagingdevices a first angle about the first rotational axis; controlling thefirst and second imaging devices to collect the first and second sets ofimage data; and merging the first and second sets of imaging dataaccording to the angle between the optical axis of the first imagingdevice and the scanning plane of the second imaging device, wherein thefirst angle is between 0° and 360°.
 13. The three-dimensional imagingapparatus of claim 12, further comprising a second motor having a secondrotational axis along a third direction and configured to rotate thefirst and second imaging devices about the second rotational axis, thethird direction being perpendicular to the first and second directions.14. The three-dimensional imaging apparatus of claim 13, wherein theprocessor is configured to execute the computer-executable instructionsto control the second motor to rotate the first and second imagingdevices a second angle about the second rotational axis, wherein thesecond angle is between 0° and 360°.
 15. A method of controlling athree-dimensional imaging apparatus having a first imaging device and asecond imaging device disposed therein, the method comprising: rotatingthe first and second imaging devices by a first motor about a firstrotational axis, the first imaging device having an optical axisextending along a first direction, the second imaging device having ascanning plane extending along a second direction, the optical axis ofthe first imaging device and the scanning plane of the second imagingdevice forming a first angle between 0° and 360; starting the first andsecond imaging devices to collect a first and second sets of image data,respectively; receiving the first set of image data from the firstimaging device and the second set of image data from the second imagingdevice; and merging the first and second sets of image data according inpart to the angle between the optical axis of the first imaging deviceand the scanning plane of the second imaging device.
 16. The method ofclaim 15, further comprising: a) sending an instruction to the firstmotor indicating a first expected rotational angle; b) rotating thefirst and second imaging devices from a first position to a secondposition according to the instruction; c) determining a first rotationalangle between the first position and the second position; and d)determining a first error between the first expected rotational angleand the first rotational angle.
 17. The method of claim 16, furthercomprising: carrying out a)-d) for a plurality of iterations; anddetermining a statistical distribution of the first error.
 18. Themethod of claim 16, further comprising: determining a first coordinateof a calibration object based on at least one of the first or secondsets of image data when the first and second imaging devices are at thefirst position; determining a second coordinate of the calibrationobject based on at least one of the first or second sets of image datawhen the first and second imaging devices are at the second position;and determining the first rotational angle based on the first coordinateand the second coordinate of the calibration object.
 19. The method ofclaim 15, wherein the three-dimensional imaging apparatus furthercomprises a second motor configured to rotate the first and secondimaging devices for a second angle about a third axis that isperpendicular to the first and second axes, the method furthercomprising: e) sending an instruction to the second motor indicating asecond expected rotational angle; f) rotating the first and secondimaging devices from a third position to a fourth position according tothe instruction; g) determining a second rotational angle between thethird position and the fourth position; and h) determining a seconderror between the second expected rotational angle and the secondrotational angle, wherein the second angle is between 0° and 360°. 20.The method of claim 19, further comprising: carrying out e)-h) for aplurality of iterations; and determining a statistical distribution ofthe second error.
 21. The method of claim 19, further comprising:determining a third coordinate of a calibration object based on at leastone of the first or second sets of image data when the first and secondimaging devices are at the third position; determining a fourthcoordinate of the calibration object based on at least one of the firstor second sets of image data when the first and second imaging devicesare at the fourth position; and determining the second rotational anglebased on the third coordinate and the fourth coordinate of thecalibration object.